The present invention relates to an image processing system capable of applying a good texture such as a blur and, more particularly, to an image processing apparatus and method applicable to an advanced-function digital camera serving as a virtual camera, and a computer-readable program storage medium used in the apparatus and method.
With the recent spread and functional advancement of home or personal computers (PCs), personal digital cameras have been developed to take the place of conventional cameras using a silver halide film. Also, higher-grade digital cameras having a CCD of 1,000,000 pixels or more are available.
This high-resolution digital camera is optimal for entertainment purposes: an image can be displayed on a high-resolution display of VGA or higher, or printed by a large-size printer for A4 size or larger.
At the same time, an image processing technique, which has conventionally been used in only special study purposes, movie production, and printing jobs, can be realized by a low-cost home computer along with downsizing by computer techniques, reduction in power consumption, and improvement of image processability.
Computers (CPUs), which were conventionally high-cost, large-size machines, have been downsized and reduced in power consumption and can now be incorporated in digital cameras.
In the digital camera, as a high-resolution CCD of, e.g., 800,000 or 1,300,000 pixels is downsized from ½″ to ¼″, the lens is also downsized to increase the F-number and shorten the focal length.
Under this circumstance, even the digital camera can have a large focal depth and are can obtain with it an in-focus image from a near distance to a far distance without any high-cost, high-precision AF function.
On the other hand, with the digital camera one can only take a similar image regardless of who the photographer is.
More specifically, a camera using a silver halide film, such as a relatively expensive single-lens reflex camera, can make an object of interest appear to float from the background or blur the periphery by arbitrarily selecting the F-number and shutter speed. Such enjoyment unique to photographing is impossible in a small-size digital camera.
However, these problems can be solved by an image processing technique.
For example, “Modeling of Blurring in Human Eyes and Binocular Stereoscopic Display by Modeling” (Kaneko et al.), 1990 (the 40th) Conference of Information Processing Society of Japan, pp. 109-110 discloses the following study. Based on depth information of a computer image, the visual blur function at the retinal position where the image is formed on the eye is approximated to a Gaussian distribution to blur the computer image.
In the field of computer image processing, filtering is often adopted in which the background is blurred using a high-pass filter, and the object image is sharpened using a low-pass filter.
Filtering using a high-pass filter is averaging as one means of image processes in which color information of pixels around a pixel of interest are averaged and replaced with the pixel of interest.
Filtering using a low-pass filter is contrast emphasis.
Further, Jpn. Pat. Appln. KOKAI Publication No. 6-118473 discloses a function of calculating inside a camera so as to apply the blur even by a small-size camera and outputting a warning, or a function of adjusting the object distance of the camera so as to apply a blur.
According to Jpn. Pat. Appln. KOKAI Publication No. 7-21365, image data, depth data, and condition information of a camera actually used to photograph an object such as the F-number and focal length are input. At the same time, a virtual image sensing parameter set by a user is separately set. A blur parameter is calculated from the input values. The calculated blur parameter is compared with the actual blur corresponding to the set value of the camera information such as the F-number and focal length to selectively use a low-pass filter and high-pass filter. Then, a new image is output to an image memory upon image processing.
According to Jpn. Pat. Appln. KOKAI Publication No. 9-181966, an image sensed using a pair of image sensing lenses having a parallax is input, and distance information is calculated based on the information. Blur parameters including any one of the F-number, f-number, and focal point position are selected to apply a blur effect.
Any of the conventional methods applies a blur by image processing. This is basically equivalent to a conventional method of filtering a computer image having data in the direction of depth. Hence, the above-described methods cannot apply any good texture to an image having a long focal length by a future micro-digital camera.
For example, in Jpn. Pat. Appln. KOKAI Publication No. 7-21365, information about any one of the F-number, f-number, and focal point position of a camera actually used to photograph an object is supplied. The blur characteristic is calculated using the set F-number, f-number, or focal point position, and a new image is produced by image processing selectively using a low-pass filter and high-pass filter. However, the actual image is blurred, so no sharp image can be obtained from this image by a low-pass filter.
When the camera uses a zoom lens, it is difficult to always supply information about any one of the F-number, f-number, and focal point position.
Such information depends on the system or camera in use, and cannot be generalized and hardly attains compatibility.
An image is blurred selectively using a low-pass filter and high-pass filter, so no good texture can be obtained.
In Jpn. Pat. Appln. KOKAI Publication No-9-181966, an image sensed using a pair of image sensing lenses having a parallax is input, and distance information is calculated based on the information. The blur parameter is selected to apply the blur effect. This satisfies the above requirement. However, since this blur parameter includes any one of the F-number, f-number, and focal point position, camera information is also necessary. The camera information is not compatible and general, and the blur does not give a good texture.
The present inventors have studied in pursuit of good textures sought for current digital cameras for long and found the following functions and requirements that are necessary for next-generation digital cameras.                The camera has high resolution with a sufficient number of pixels, e.g., 800,000 pixels or more.        The camera can focus from infinity to a relatively near distance.        The camera has information in the direction of depth of an object.        Pieces of depth information are collected in units of objects in order to remove noise.        Camera information except for an image and depth information is unnecessary.        The field angle and lens information are set only in setting a virtual camera.        The texture applied by image processing includes a blur, color, and reflectivity.        The blur is determined based on the lens theory of the camera, and is natural.        The blur, color, and reflectivity can be adjusted.        
Unless these problems are solved, the texture of the digital camera is practically difficult to improve.
To satisfy these conditions, realization of a digital camera capable of applying a texture is demanded.